This disclosure concerns automated quality grading for digital images assembled from image segments that are joined at overlaps along one or more margins.
In the processing of microscopic images of pathology and histology samples on tissue sample slides, digital images of small adjacent areas over the surface of the sample can be captured at high magnification and/or resolution, and then displayed and managed as parts of a larger image that the user navigates using a digital display device. Small squares or strips or other image shapes that were separately captured, are image segments that can be aligned relative to one another to make up a mosaic. Each square or strip or other separately captured image of any arbitrary shape is a “tile” in the mosaic.
The whole mosaic of separately captured images, or some part of the mosaic encompassing a group of adjacent tiles, might be shown on the display device at relatively low resolution. The user preferably can zoom to a higher resolution view wherein the display encompasses an image segment of one mosaic tile, or a still higher resolution view of an area smaller than a tile. The user can navigate by panning the display across a boundary between adjacent image segments obtained from two different image acquisition operations, namely across the border between tiles.
The separately captured image areas need not be small or regularly shaped or in regular positions in an array. Larger portions such as full sample height, partial width image blocks, laterally and/or longitudinally abutted blocks such as quadrants, blocks of different sizes or shapes, are all possible. The image areas might be acquired as snapshot images or as the result of scanning over elongated strips over time using a line scan technique. For purposes of discussion in this disclosure, separately acquired image segments of any shape or size generally will be called tiles.
Inasmuch as the images are collected in separate image collection operations, it is advantageous to align the images accurately to enable smooth navigation spanning the image boundaries. This can be done by lining up edges or shapes that span across the border between two images or tiles. In one technique, the images are captured with overlapping margins. The adjacent images contain the same content in the margins. The adjacent tiles are aligned by adjusting their relative positions until the features shown in the two overlapped margins are registered. It is possible to slice an abrupt transition at which the pixel values for one adjacent tile change to the pixel values of the next adjacent tile. For a smooth transition, the pixel data of the image segments can be merged across the margin by a weighted averaging of pixel data values or similar merging techniques, generally known as “stitching” together the adjacent images. In the final mosaic image, there is one X-Y field of pixel data values, although in the originally captured tiles, the overlapping margins provided redundant images of the overlapping margins.
A digital pathology system advantageously emulates certain aspects of sample handling and microscope manipulation that are familiar to pathologists. In traditional manual operations, samples mounted on glass slides are placed on the stage of an optical microscope configured for viewing at some selected magnification that may be great or small. The view of the slide seen through the microscope at high magnification is a view of a small targeted area. The area selected as the target can be translated across the slide by adjusting the X-Y position of the stage and/or by moving the slide relative to the stage. The field of view can be made larger or smaller by changing the extent of magnification, typically by rotating a mounting so as to substitute a different objective lens in the optical path. Zooming in or out in a digital display emulates changing magnification. Panning a digital display emulates moving the X-Y position of a slide on a microscope stage.
In some digital pathology systems, one or more cameras coupled to the microscope optics collects a stop image frame, and the stage is advanced in an X and/or Y direction to present a next area adjacent to the previously imaged area and another image is collected. In such an arrangement, the respective tile images can be stitched or merged together at top, bottom and lateral side edges. In a system having a linear scanning charge-coupled-device sensor (CCD), the sensor collects an elongated line image while being moved in a direction more or less perpendicular to its direction of sensor elongation, at a sampling period that produces a predetermined pixel spacing. The necessary merging or stitching in such a system may be limited to two opposite edges of a strip (or one edge for the extreme top and bottom strips).
Another degree of freedom in a microscope is in the Z direction, perpendicular to the X-Y plane of the sample on the stage. The optics of a microscope are such that features of a sample appear to be in focus when located at a specific distance from the objective lens or lens array, i.e., at a correct focal distance. The features become blurred or indistinct if nearer to the lens or farther away. There is a certain depth of field or range of distances in the Z direction wherein features appear to be in focus. That depth of field is affected by the lens aperture size.
The surface at which features appear to be in focus is generally shaped as a sphere. The sphere may have a relatively longer or shorter radius compared to the format size of the image, but is nevertheless a sphere. The pathology or histology sample on the sample slide, however, has a surface that is generally along a plane rather than a spherical surface. Image features in the center of the field of view may be in focus when features at the periphery are not in focus (or vice versa). The sample itself may be thicker or thinner in different areas, such that the topography of the sample places lower or higher parts on the surface of the sample at the optimal focus distance or above or below the optimal distance. If the sample or the stage is tilted, the quality of the image may vary continuously from one part of the image to another. The optics of the microscope may have aspects such as optical aberration or distortion that tend to favor the view at the center of the field of view versus the periphery. It would be advantageous to have an effective means to judge these effects.
Images have content characterized by features of various types, occurring at particular locations in the image and/or distributed across the images. Mathematical algorithms are known for application to digital images comprising arrays of pixel values, wherein the algorithms contain one or more mathematical functions that are sensitive to the presence of particular features. By operating such algorithms on pixel data, a numeric assessment is obtained that measures the extent to which the features are present, producing output values that are greater or less depending on the presence and prominence of associated features. The assessment can be localized, producing values associated with individual pixel positions or groups of pixels, or the assessment can be generalized across the image, producing one output value. Operating an algorithm that is sensitive to a feature to produce a numeric value in that way, such as assessing the level of contrast between pixels at a certain distance, is known as extracting that feature. The algorithm that does the mathematical analysis is known as a feature extraction algorithm.
One can apply such mathematical analyses to pixel image data in an effort to characterize the image quality of a digital image from a microscope or other source. However objective measures that may correlate with image quality also are affected by variations in the content of the image apart from image quality. For example, an objective measure correlating with the sharpness of an image can be obtained by integrating the pixel-to-pixel differential in luminance and/or color value between each pixel and its neighboring pixels, across an image or in a discrete area of the image (i.e., by extracting that feature). Other measures could involve assessment of the peak signal to noise ratio across the pixel data values, the range of variation in luminance or color values, a statistical analysis of pixel data, etc. These measures may correlate with image quality. For example a high pixel data differential correlates with sharp focus. But such measurements also correlate with image content. A poorly focused image of content characterized by high inherent contrast may produce a higher value according to such objective measures than an accurately focused image of content characterized by low inherent contrast.
Inasmuch as quality measurements based on feature extraction algorithms are affected by content, the results of objective measures of quality are meaningful if the image quality is a variable and the content of the image is not a variable. For this reason, imaging calibration techniques sometimes involve the use of a standardized test pattern of defined shapes and/or colors.
Autofocus techniques use objective measures comparatively, typically to compare the objective attributes of two or more images of the same content at different focal distances, so as to determine which focal distance provides the sharpest focus. Pixel data focused at a given Z distance for a microscopic image in an X-Y plane (or at least for a localized part of the image) can be processed by a feature extraction algorithm to provide a calculated numerical characterization of the image (or localized part) that correlates with image quality, especially focus accuracy. After shifting the Z distance, another pixel data image is obtained at the new distance and processed to obtain another numerical characterization from the same algorithm. The difference between the numerical characterizations is due to the shift in Z distance, because the content is the same. The numerical characterization and the Z distance correlate with the accuracy of image focus. This technique enables comparison of images at two or more distances to choose a distance that produces a relatively accurate focus.
After collecting images at a sufficient number of Z distances (a minimum of three distances), and assuming that the Z distances span the distance at which the optimal focus accuracy might be obtained, it may be possible to calculate the Z distance of the “correct” or most accurately focused plane. The focal plane is adjusted to that Z distance and a final image is collected. To accomplish this, the calculated results of the algorithm are matched to a characteristic curve wherein a peak (or valley, depending on the nature of the calculation) of the curve is considered to occur at the Z distance of optimal focus.
This process is typical of autofocus processes and relies on comparing, for different Z distances, the results of an algorithmic mathematical characterization known to correlated with image quality, especially focus accuracy. Exemplary algorithms include local spatial derivative assessments (perhaps integrated over the whole image or only at a selected discrete area), statistical measures applied to pixel data values for the image or for a discrete area, and similar measures. Additional examples are mentioned below. Focus accuracy is an important characteristic to optimize, but the same considerations apply to other characterizations that may correlate with image quality and may be affected by variables that are controllable, such as the luminance level of a bright field, which can affect quality variables such as the color gamut and relative RGB spectra obtained from color test reference patterns, ratio of peak signal to noise, comparison of central and peripheral pixel values, and other quality measures.
Instead of comparing two or more images obtained from the same content when using two or more different sets of conditions known to affect image quality (such as different Z distances known to affect focus accuracy), one might apply an algorithm to assess image quality for a single image, and obtain one or more output values. Although the value may correlate with an image quality attribute, the results have little meaning unless the image is an image of a known calibration standard (e.g., a test pattern). Even within a single digital pathology image, a sample may contain any of various tissue types and structures. An algorithm correlated with an image quality variable such as focus accuracy produces a given output value for images showing tissue types that are exactly focused but are inherently smoothly varied in appearance (such as the relatively featureless connective tissue or stroma between distinct features, for example). That given output value for inherently low-contrast image content can be comparable to the output value obtained for blurred and poorly focused tissues types that have inherent variations (such dense dermal cells, for example).
Other considerations affect focus accuracy. A sample such as a piece of biological tissue may have a topography characterized by different thickness at different points in the X-Y plane, causing some surfaces on the sample to be closer to the lens than other surfaces in the Z direction. The sample may not be mounted in a way that exposes a plane surface to view. Due to mechanical misalignment, the slide may be tilted relative to a plane tangent to the focus sphere (i.e., other than normal to the optical axis). As a result, the situation may occur wherein some image areas on a slide, and/or some X-Y points thereon, are closer to the microscope optics than other areas or points. Due to characteristics of the optics, the image may be better in some areas, such as close to an optical axis at the center of a field of view, and worse in other areas, such as proceeding out to the periphery of the field of view.
Adjusting the focus usually entails varying the relative distance from an objective lens with one or more relatively fixed lens parts, up to the surface of the sample, bringing the features of interest into the so-called focal plane. The stage holding the sample may be movable toward or away from the mounting of the objective lens, or vice versa. In a manual microscope, a control knob is used for adjusting the distance in the Z direction. In an automated scanning microscope, an electromagnetic or piezoelectric mechanism or the like, controllably adjusts the distance in the Z direction.
When viewing the sample through the microscope and adjusting for focus with a manual control knob, one typically moves the Z distance up to and beyond the point of optimal focus, and then moves back, homing in on the correct focal distance by adjusting to obtain the sharpest image available during viewing. After manually dithering through the focal distance in this way, the operator has some confidence that the sample has been viewed for all that it reveals, namely in the best focus available from the instrument. The manual homing or dithering operation may also be sufficient to pass the sample through a range of Z distances that is sufficient to exceed the range of Z distances caused by variations in topography. But if an image is collected automatically, the image is taken at a given distance in the Z direction. If one requires viewing at two or more different Z distances, then it is necessary to collect two or more digital images.
In a digital microscopy system, tissue samples are prepared in the usual way of being mounted on glass slides, but instead of having the pathologist view the samples using a manually controlled optical microscope, the slides are processed using digital cameras coupled to the microscope optics to collect microscopic pictures. Incremental stage positioning controls step the viewing area over the surface of the slides. The scanners can collect images of the sample at different resolutions or preferably, images taken at high resolution can be combined or “stitched” together to provide image files that encompass plural high resolution images. The pathologist views the digitized images of the slides on a computer workstation, using the zoom and pan functions of image display software to navigate the sample. Disclosures of collecting and stitching together high resolution images of adjacent areas for such purposes, can be found for example, in published US applications 2009/0195688—Henderson et al. and 2008/0240613—Dietz et al., the disclosures of which are hereby incorporated. Such techniques achieve many of the functions of manually controlled optical microscopes, and have additional advantages. For example, the digital data can be stored indefinitely as a permanent record. Image data can be retrieved and transmitted readily using network communications. Digital images of slides can be organized and used more efficiently than the glass slides themselves. Digital images can be navigated rapidly for X-Y position and magnification/zoom, as well as annotated and processed in various ways.
Typically only one digital image is stored per logical image segment, e.g., one image per tile in a mosaic. It would be possible in a digital pathology system to use a scanning device to record multiple images of the same area at slightly different focal distances, with the imaged surface of the sample being slightly above, slightly below and preferably just at the optimal focal distance of the microscope optics used in the scanning device. With a sufficient number of views, this could enable a person viewing images on a computer workstation to select images at slightly different focal distances in the same way that a user of an optical microscope dithers the focal distance adjusting knob to seek the distance with the best focus. However, data processing and data storage needs would be multiplied to obtain, store and manage each high resolution image.
Accordingly, preliminary steps are undertaken to adjust for focus, preferably when collecting every logical image segment but optionally at some other schedule. The slide is imaged from the focal distance that is chosen during preliminary focusing operations. Instead of recording multiple images at different focal distances (wherein some of the images inherently would be taken at an incorrect focal length), an autofocusing control is employed to make the necessary focus adjustments before the image of a tile or other segment is recorded and stored. The plan is to record one image per final image capture operation. The autofocusing control selects the optimal focus distance at which the image is captured. An X-Y stage positioning control advances the field of view to an next position on the slide. The process is repeated to pass over and collect snapshot frame images of the entire slide or the full area of the sample, or of a selected area to be imaged.
Autofocusing controls in digital imaging operate by numerically analyzing the pixel data with an algorithm that measures a total amount of contrast in luminance or color values between adjacent pixel positions. A higher total value for the contrast measurements indicates that an image is in better focus, other things being equal. Autofocus controls typically attempt to compare alternatively focused versions of the same image content, such as the same content focused at slightly different focal distances. It is possible to provide a numerical measure of total contrast in an image, but one cannot meaningfully determine focus quality independent of the content of an image because the results of numerical measures, such as an integrated total of the local contrast through the image, vary with image content as well as with the accuracy of focus.
An exemplary autofocus control is disclosed in U.S. Pat. No. 7,576,307—Yazdanfar et al., hereby incorporated by reference. A predictive autofocus control is disclosed in US Publication 2009/0195688—Henderson et al., mentioned and incorporated above. These controls use primary and secondary imaging sensors, and a Z positioning control for altering the focal length. An object is to compare the accuracy of focus at two or more different focal distances, and to find the distance at which the focus is best, whereupon an image at that distance is taken and stored. The process involves comparing a measure of focus quality for two or more differently focused images of the same image content. In that situation, a numerical measure of focus quality is meaningful (such as a sum of local contrast values for all pixel positions). One can conclude that the focus distance that produces a higher total from the measure of contrast (or other focus quality measure) is closer to the best possible focal distance than the distance that produces a lower total from the same measure.
The collected images become the tiles or local frame areas. These areas typically overlap somewhat in zones where the content of the tiles is aligned and the pixel values are knitted or “stitched” to merge over the transitions with other adjacent tiles so as to form a composite image of the sample. Examples of image alignment and stitching can be found in U.S. Pat. No. 6,785,427—Zhou and published application US 2008/0240613—Dietz, which are hereby incorporated in the present disclosure by reference.
When collecting digital images of image segments adjacent to or spaced from a given position at which autofocusing and Z position controls were used to select a best focal distance, one can assume that the same distance is the best distance for all other image segment positions across the sample. At the other extreme, it is possible to repeat the autofocusing and Z position selection steps to attempt to obtain a best focal distance independently for each image segment. Another alternative is to obtain a best Z position for a given image segment and to use that Z position for all image segments within a certain X and/or Y distance of the given image segment, repeating the Z position selection at intervals of a certain number of image segments, or when changing from one row or column of image segments to a next, etc. In published application US 2006/0204072—Wetzel, which is hereby incorporated, plural tiles at spaced points are tested for optimal Z positions for best focus at each point. A reference plane is then fitted as nearly as possible to intersect the best focus distances, and the Z positions for each tile across the specimen are chosen as the corresponding positions on the fitted plane.
A best focus Z position may be determined for a reference image segment and used for some larger area encompassing adjacent image segments. For example, an image segment at the center of a specimen or a zone on the specimen, can be used as a reference image segment for which an optimum Z position is determined. That Z position can be chosen after a relatively involved and careful autofocusing procedure, and used for a group of remaining image segments that are stitched together into an image or portion of an image. Alternatively, the Z position of the reference image segment can be merely a starting position when undertaking a new autofocusing procedure for each next tile, optionally using a procedure that is less involved. It remains possible that the best focal distances for different image segments or distinct areas within the segments may be at different focal distances, considering the mounting of the slide, the topography of the sample and other factors. What is needed is a way to compare and to indicate or otherwise respond to a situation in which the focus quality varies between the reference image segment and other image segments in the group wherein the Z position might or might not have been determined independently, and the image segments are stitched together into a larger mosaic or montage image.
Image assessment algorithms are known, but generally are most useful to compare alternative images of the same image content at different focal distances. A challenge is presented when attempting to assess image quality independent of image content, for example to grade the quality of focus for various images showing different samples or different areas on a given sample.
The foregoing discussion and examples are applied only to focal distance and focus quality issues. Focus is one example of the more general issue of choosing among the alternative conditions that are used to collect a digital image of a sample, when differences in the conditions affect image quality. For example, it may be possible to select among alternative conditions of front and/or rear lighting or lighting amplitude, illumination spectra, polarization conditions, image collection time, aperture size and depth of field, etc. Typical solutions employ nominal conditions or in the case of autofocus use a controller to select perceived optimal conditions, but can result in some variation in the depth of the focal plane from one image segment to another and some variation in the quality of the focus. This disclosure is primarily exemplified using focus accuracy as the image quality criterion of interest and Z position as the variable that affects image quality as measured by contrast. The disclosure is not limited only to that quality criterion, or only to that variable or technique for measuring image quality. The disclosure is likewise applicable to other quality criteria, other variables that affect focus and/or other quality criteria, and other possible measures by which focus or other quality criteria might be measured.
Using focus as a representative image quality criterion, there are a number of focus assessment image processing algorithms that can produce a numerical measure of focus quality that may not indicate a level of quality in an absolute sense but at least varies with quality in a way the allows two pictures of the same image content to be meaningfully compared. In “Autofocusing in Computer Microscopy: Selecting the Optimal Focus Algorithm,” Y. Sun et al., Microscopy Research and Technique 65:139-149 (2004), the following algorithms are compared:
Derivative Based Algorithms:                Thresholded Absolute Gradient (Santos et al., 1997)        Squared Gradient (Santos et al., 1997)        Brenner Gradient (Brenner et al., 1971)        Tenenbaum Gradient (Tenengrad) (Yeo et al., 1993, Krotov, 1987)        Sum of Modified Laplace (Nayar and Nakagawa, 1994)        Energy Laplace (Subbarao et al., 1993)        Wavelet Algorithm (Yang and Nelson, 2003)        Wavelet Algorithm W2 (Yang and Nelson, 2003)        Wavelet Algorithm W3 (Yang and Nelson, 2003)        
Statistical Algorithms:                Variance (Groen et al., 1985, Yeo et al., 1993)        Normalized Variance (Groen et al., 1985, Yeo et al., 1993)        Autocorrelation (Vollath, 1987, 1988)        Standard Deviation-Based Correlation (Vollath, 1987, 1988)        
Histogram-Based Algorithms                Range Algorithm (Firestone et al., 1991)        Entropy Algorithm (Firestone et al., 1991)        
Intuitive Algorithms                Thresholded Content (Groen et al, 1985, Mendelsohn and Mayall, 1972)        Thresholded Pixel Count (Green et al., 1985)        Image Power (Santos et al., 1997)        
Such image processing algorithms are useful to obtain an objective measure of pixel data characteristics such as the level of contrast found between pixels or groups of pixels in an image, which measure correlates with focus accuracy but also correlates with image structural characteristics, i.e., image content. Such image processing algorithms can compare image quality characteristics in a meaningful way when comparing alternative images containing the same content (i.e., when all aspects of the comparison are the same, except for the focus or other aspect of image quality). A numeric algorithm produces different objective scores for different types of image content. The algorithm may produce a lower objective score of contrast (or other variable value associated with quality or focus accuracy) for some types of content compared to other types of content, even when the quality is actually better in the image whose content produces the lower objective score.
What is needed is a way to make an automated assessment of image quality, especially focus accuracy, that is independent of image content or is as dependable as a comparison of the same image content in two alternative pictures of the same image content, but without the complications that are typical of autofocusing techniques wherein two or more images of the same image frames are collected at different Z axis distances, and compared to determine whether one or the other produces an image of better quality.